On the validity of measuring change over time in routine clinical assessment: a close examination of item-level response shifts in psychosomatic inpatients

Objective: Significant life events such as severe health status changes or intensive medical treatment often trigger response shifts in individuals that may hamper the comparison of measurements over time. Drawing from the Oort model, this study aims at detecting response shift at the item level in psychosomatic inpatients and evaluating its impact on the validity of comparing repeated measurements. Study design and setting: Complete pretest and posttest data were available from 1188 patients who had filled out the ICD-10 Symptom Rating (ISR) scale at admission and discharge, on average 24 days after intake. Reconceptualization, reprioritization, and recalibration response shifts were explored applying tests of measurement invariance. In the item-level approach, all model parameters were constrained to be equal between pretest and posttest. If non-invariance was detected, these were linked to the different types of response shift. Results: When constraining across-occasion model parameters, model fit worsened as indicated by a significant Satorra–Bentler Chi-square difference test suggesting potential presence of response shifts. A close examination revealed presence of two types of response shift, i.e., (non)uniform recalibration and both higher- and lower-level reconceptualization response shifts leading to four model adjustments. Conclusions: Our analyses suggest that psychosomatic inpatients experienced some response shifts during their hospital stay. According to the hierarchy of measurement invariance, however, only one of the detected non-invariances is critical for unbiased mean comparisons over time, which did not have a substantial impact on estimating change. Hence, the use of the ISR can be recommended for outcomes assessment in clinical routine, as change score estimates do not seem hampered by response shift effects.

Change of perspective: a measurable and desired outcome of chronic disease self-management intervention programs that violates the premise of preintervention/postintervention assessment

Objective: To determine if participants in chronic disease self-management courses have a change of perspective of their health status (a response shift), and if this is measurable with a paper-based questionnaire.

Methods: Nine items were developed to measure potential benefits of self-management courses. These were based on the constructs of a previous questionnaire, the Health Education Impact Questionnaire (HEI-Q). Cognitive interviews elicited spontaneous statements about the reasons for paper-based answers. Sensitivity, specificity, and overall accuracy of items were calculated using the interview as a relative gold standard. Response shift can be negative (i.e., after the course, participants realize that, before the course, they were worse than they thought they were), positive (i.e., participants now realize they were better than they thought they were), or absent (no change).

Results: Interviews (n = 39) reflected that true response shift occurred in approximately half the replies to questionnaire items. Of these, 31% were negative response shift, 20% were positive response shift. Response shift was absent in 32% of replies. Presence or absence of response shift could not be determined in 17% of replies across items. Significant concordance between questionnaires and cognitive interviews (average overall accuracy 0.79) indicated that the HEI-Q Perspective questionnaire detects response shift in participants of self-management courses. The questionnaire revealed that 87% of participants had response shift in at least 1 item.

Conclusion: This study suggests that preintervention/postintervention assessments of interventions such as self-management courses are confounded by a change in perspective of a large proportion of respondents. It also indicates response shift is a valuable outcome of self-management courses that can be measured with a paper-based questionnaire.

Tests of measurement invariance failed to support the application of the "then-test"

Objective
The use of then-test (retrospective pre-test) scores has frequently been proposed as a solution to potential confounding of change scores because of response shift, as it is assumed that then-test and post-test responses are provided from the same perspective. However, this assumption has not been formally tested using robust quantitative methods. The aim of this study was to compare the psychometric performance of then-test/post-test with traditional pre-test/post-test data and assessing whether the resulting data structures support the application of the then-test for evaluations of chronic disease self-management interventions.

Study Design and Setting
Pre-test, post-test, and then-test data were collected from 314 participants of self-management courses using the Health Education Impact Questionnaire (heiQ). The derived change scores (pre-test/post-test; then-test/post-test) were examined for their psychometric performance using tests of measurement invariance.

Results
Few questionnaire items were noninvariant across pre-test/post-test, with four items identified and requiring removal to enable an unbiased comparison of factor means. In contrast, 12 items were identified and required removal in then-test/post-test data to avoid biased change score estimates.

Conclusion
Traditional pre-test/post-test data appear to be robust with little indication of response shift. In contrast, the weaker psychometric performance of then-test/post-test data suggests psychometric flaws that may be the result of implicit theory of change, social desirability, and recall bias.

Comparison of the pre-post and transition question assessments in a health education setting

Objective : This study examined the classical pre-intervention/post-intervention assessment (pre–post) and the single post-intervention transition question assessment (transition question) to determine how well these methods reflected qualitative interview–based participant-reported outcomes from chronic disease self-management education programs (CDSMEPs).

Study Design and Setting : A mixed-method qualitative and quantitative approach was applied in 25 interviews with participants recruited from CDSMEPs within Australia. Qualitative interviews with participants were used as a relative “gold standard” and compared with questionnaire-based pre–post and transition question assessments.

Results : Comparison of the two questionnaire-based assessments showed that most of the individual paired responses were discordant (61%). Using participant's qualitative narratives as a “true” indicator, the pre–post assessment was found on more occasions to be discordant with participant-reported outcomes than the transition question. The origin of the inconsistency was largely because of a change in respondents' perspective that had occurred after pretest, which was mediated by CDSMEPs' experiences and insights.

Conclusion : This study suggests that the pre–post assessment has poor validity for the assessment of health education program outcomes. Alternative assessments, such as the transition question, may result in a more accurate reflection of the impact of such programs on participants.

The inclusion of ‘then-test’ questions in post-test questionnaires alters post-test responses: a randomized study of bias in health program evaluation

Objectives Program evaluations are frequently based on ‘then-test’ data, i.e., pre-test collected in retrospect. While the application of the then-test has practical advantages, little is known about the validity of then-test data. Because of the collection of then-test in close proximity to post-test questions, this study was aimed at exploring whether the presence of then-test questions in post-test questionnaires influenced subjects’ responses to post-test.
Patients and methods To test the influence of then-test questions, we designed a randomized three-group study in the context of chronic disease self-management programs. Interventions had comparable goals and philosophies, and all 949 study participants filled out identical Health Education Impact Questionnaires (heiQ) at pre-test. At post-test, participants were then randomized to one of the following three groups: Group A responded to post-test questions only (n = 331); Group B filled out transition questions in addition to post-test (n = 304); and Group C filled out then-test questions in addition to post-test (n = 314).
Results Significant post-test differences were found in six of eight heiQ scales, with respondents who filled out then test questions reporting significantly higher post-test scores than respondents of the other groups.
Conclusions This study provides evidence that the inclusion of then-test questions alters post-test responses,
suggesting that change scores based on then-test data be interpreted with care.

Measurement issues in the evaluation of chronic disease self-management programs

Objectives: 

Self-perceived distress and impairment in problem gamblers: a study of pre- to post-treatment measurement invariance

Gambling help services typically evaluate treatment outcomes using self-reported responses and measurements. However, gamblers’ conceptualisations and prioritisations with respect to these measurements may shift over time. Thus, changes in the self-reported responses may not always reflect true change in the individuals. This study investigated for response shift in self-report measures of psychological distress and impairment in 293 help-seeking problem gamblers. We used confirmatory factor analysis to model data structures from pre-treatment to post-treatment. The findings indicated that a response shift had occurred. Two items became less important and one item became more important in measuring psychological distress. Measurement invariance was achieved for the complete set of items for impairment. These findings provide a more in-depth understanding of the nature of self-report outcomes in otherwise routinely collected data.

Self-perceived distress and impairment in problem gamblers : a study of pre- to post-treatment measurement invariance

Gambling help services typically evaluate treatment outcomes using self-reported responses and measurements. However, gamblers’ conceptualisations and prioritisations with respect to these measurements may shift over time. Thus, changes in the self-reported responses may not always reflect true change in the individuals. This study investigated for response shift in self-report measures of psychological distress and impairment in 293 help-seeking problem gamblers. We used confirmatory factor analysis to model data structures from pre-treatment to post-treatment. The findings indicated that a response shift had occurred. Two items became less important and one item became more important in measuring psychological distress. Measurement invariance was achieved for the complete set of items for impairment. These findings provide a more in-depth understanding of the nature of self-report outcomes in otherwise routinely collected data.

Population consequences of a predator-induced habitat shift by trout in whole-lake experiments

In a replicated whole-lake experiment, we (a) tested for the existence of a flexible habitat shift in response to predator presence in age-0 rainbow trout (Oncorhynchus mykiss) at risk of cannibalism and (b) evaluated the population-level consequences of habitat shifts in terms of growth and survival over their first growing season. Daphnid food and adult trout predators were substantially more abundant in pelagic than in littoral habitats. Age-0 trout used all habitats in populations without adult trout predators, whereas age-0 trout were observed only in the less profitable littoral habitat in populations with adult trout. Consequently, mean fall mass of age-0 trout in the presence of predators was almost half that observed in populations without adult trout. Despite the shift in habitat use, age-0 trout experienced 90% mortality when adult trout predators were present, in comparison to only 36% mortality when absent. We conclude that the commonly observed habitat shifts by fish at risk of predation, observed at smaller scales, do in fact occur at the whole-system scale over long time intervals. These results suggest that fish are able to perceive risk at large spatial scales and thus take advantage of profitable (but normally risky) habitats when predators are absent, or move to less profitable refuge habitats when predators are present.

Characterization of the MEK5-ERK5 module in human neutrophils and its relationship to ERK1/ERK2 in the chemotactic response

The role of the extracellular signal-regulated kinase (ERK) 1 and ERK2 in the neutrophil chemotactic response remains to be identified since a previously used specific inhibitor of MEK1 and MEK2, PD98059, that was used to provide evidence for a role of ERK1 and ERK2 in regulating chemotaxis, has recently been reported to also inhibit MEK5. This issue is made more critical by our present finding that human neutrophils express mitogen-activated protein (MAP) kinase/ERK kinase (MEK)5 and ERK5 (Big MAP kinase), and that their activities were stimulated by the bacterial tripeptide, formyl methionyl-leucyl-phenylalanine (fMLP). Dose response studies demonstrated a bell-shaped profile of fMLP-stimulated MEK5 and ERK5 activation, but this was left-shifted when compared with the profile of fMLP-stimulated chemotaxis. Kinetics studies demonstrated increases in kinase activity within 2 min, peaking at 3–5 min, and MEK5 activation was more persistent than that of ERK5. There were some similarities as well as differences in the pattern of activation between fMLP-stimulated ERK1 and ERK2, and MEK5-ERK5 activation. The up-regulation of MEK5-ERK5 activities was dependent on phosphatidylinositol 3-kinase. Studies with the recently described specific MEK inhibitor, PD184352, at concentrations that inhibited ERK1 and ERK2 but not ERK5 activity demonstrate that the ERK1 and ERK2 modules were involved in regulating fMLP-stimulated chemotaxis and chemokinesis. Our data suggest that the MEK5-ERK5 module is likely to regulate neutrophil responses at very low chemoattractant concentrations whereas at higher concentrations, a shift to the ERK1/ERK2 and p38 modules is apparent.

Detecting range shifts among Australian fishes in response to climate change

One of the most obvious and expected impacts of climate change is a shift in the distributional range of organisms, which could have considerable ecological and economic consequences. Australian waters are hotspots for climate-induced environmental changes; here, we review these potential changes and their apparent and potential implications for freshwater, estuarine and marine fish. Our meta-analysis detected 300 papers globally on 'fish' and 'range shifts', with ∼7% being from Australia. Of the Australian papers, only one study exhibited definitive evidence of climate-induced range shifts, with most studies focussing instead on future predictions. There was little consensus in the literature regarding the definition of 'range', largely because of populations having distributions that fluctuate regularly. For example, many marine populations have broad dispersal of offspring (causing vagrancy). Similarly, in freshwater and estuarine systems, regular environmental changes (e.g. seasonal, ENSO cycles not related to climate change) cause expansion and contraction of populations, which confounds efforts to detect range 'shifts'. We found that increases in water temperature, reduced freshwater flows and changes in ocean currents are likely to be the key drivers of climate-induced range shifts in Australian fishes. Although large-scale frequent and rigorous direct surveys of fishes across their entire distributional ranges, especially at range edges, will be essential to detect range shifts of fishes in response to climate change, we suggest careful co-opting of fisheries, museum and other regional databases as a potential, but imperfect alternative. © 2011 CSIRO Open Access.

Australian native gardens: Is there scope for a community shift?

The negative impacts of urbanization on biodiversity are well known, and the use of native vegetation in private gardens and streetscapes have been shown to improve the species richness and abundance of native wildlife, thereby improving the biodiversity of the local area. This study poses the question of whether the general public is interested in planting native species, to determine whether a cultural shift in garden planting style is feasible. A total of 3707 questionnaires relating to nature in the backyard were delivered to residents in metropolitan Melbourne, Australia with 417 responses received (11.2% response rate). The results indicate that the public perception of the aesthetic appeal of native gardens is fairly positive and that Melbournians have considerable interest in planting native species in residential gardens and that a large number would like wildlife in their yards. The paper concludes that there is scope to encourage the use of native plants in residential landscaping.